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REQUEST USER TO INPUT THE NAMES OF ^ S1 
EACH PARTICIPANT 



ALLOCATE A UNIQUE IDENTIFICATION 
NUMBER TO EACH PARTICIPANT AND STORE 
RELATIONSHIP BETWEEN IDENTIFICATION 
NUMBERS AND NAMES IN THE MEETING 
ARCHIVE DATABASE 
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REQUEST USER TO INPUT THE NAMES OF ^ S3 
EACH OBJECT 



ALLOCATE A UNIQUE IDENTIFICATION 
NUMBER TO EACH OBJECT AND STORE 
RELATIONSHIP BETWEEN IDENTIFICATION 
NUMBERS AND NAMES IN THE MEETING 
ARCHIVE DATABASE 
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WHOM A MODEL IS NOT ALREADY STORED 



STORE THE DATA INPUT BY THE USER 
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RENDER EACH MODEL INPUT BY THE USER 
TO DISPLAY THE MODEL AND REQUEST THE 
USER TO IDENTIFY FEATURES IN EACH 
MODEL 
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TO ENABLE THE CAMERAS TO BE 
CALIBRATED 



PERFORM PROCESSING TO CALIBRATE THE 
CAMERAS AND STORE CAMERA 
CALIBRATION DATA 



REQUEST USER TO PERFORM STEPS TO 
ENABLE THE POSITION AND ORIENTATION 
OF EACH OBJECT TO BE CALIBRATED 
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POSITION AND ORIENTATION OF EACH 
OBJECT AND STORE OBJECT CALIBRATION 
DATA 
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TIME 
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PARTICIPANTS HEAD FOR EACH CAMERA 
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FIT THE PARTICIPANT'S HEAD MODEL TO 
THE IMAGE DATA FROM EACH CAMERA TO 

DETERMINE AN ESTIMATE OF THE 
ORIENTATION OF THE PARTICIPANT'S HEAD 
FOR EACH CAMERA 
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OUTPUT AUDIBLE SIGNAL TO INDICATE 
THAT MEETING CAN COMMENCE 
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READ CURRENT ESTIMATES OF THE 3D 
POSITION AND ORIENTATION OF THE 
PARTICIPANTS HEAD 
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RENDER THE MODEL OF THE PARTICIPANT'S 
HEAD USING THE CURRENT ESTIMATES AND 
THE CAMERA CALIBRATION DATA 
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EXTRACT CAMERA IMAGE DATA FROM AN 
AREA SURROUNDING THE EXPECTED 
POSITION OF EACH HEAD FEATURE 
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PERFORM MATCHING OF THE RENDERED 
HEAD MODEL AND CAMERA IMAGE DATA TO 
FIND THE CAMERA IMAGE DATA WHICH 
BEST MATCHES THE RENDERED HEAD 
MODEL 
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DETERMINE THE 3D POSITION AND 
ORIENTATION OF THE PARTICIPANT'S HEAD 
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GENERATE A VIEWING PARAMETER FOR 
EACH PARTICIPANT DEFINING AT WHOM 
OR WHAT THE PARTICIPANT IS LOOKING 





STORE THE VIEWING PARAMETER OF EACH 
SPEAKING PARTICIPANT FOR SUBSEQUENT 
ANALYSIS 




PERFORM SPEECH RECOGNITION 
PROCESSING USING THE SPEECH 
PARAMETERS OF THE SPEAKING 
PARTICIPANT(S) TO GENERATE TEXT DATA 
FOR EACH SPEAKING PARTICIPANT 
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DETERMINE WHICH IMAGE DATA IS TO 
BE STORED IN THE MEETING ARCHIVE 
DATABASE 



ENCODE THE SELECTED VIDEO DATA AND 
SOUND DATA AS MPEG 2 DATA AND STORE 
IN THE MEETING ARCHIVE DATABASE 
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STORE THE TEXT DATA IN THE MEETING ^ S92 
ARCHIVE DATABASE 



STORE THE VIEWING PARAMETER FOR 
EACH PARTICIPANT IN THE MEETING 
ARCHIVE DATABASE 
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PARAMETER VALUES FOR EACH 
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SPEAKING TO GENERATE A VIEWING 
HISTOGRAM 



S98 



STORE THE VIEWING HISTOGRAM(S) IN THE 
MEETING ARCHIVE DATABASE, LINKED TO 
THE ASSOCIATED TEXT 



CORRECT DATA STORED IN THE 
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READ THE CURRENT ORIENTATION OF THE 
NEXT PARTICIPANT'S HEAD 
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VIEWING RAY OF THE PARTICIPANT AND 
EACH NOTIONAL LINE CONNECTING THE 
HEAD OF THE PARTICIPANT WITH THE HEAD 
OF ANOTHER PARTICIPANT 
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THE PARTICIPANT CONNECTED BY 
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VIEWING RAY 
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FOR THE PARTICIPANT TO THE 
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OBJECT TO THE PARTICIPANT 
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VIEWING RAY 
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PROCESS DATA FROM THE MICROPHONE 
ARRAY TO DETERMINE THE DIRECTION(S) 
FROM WHICH THE SPEECH IS COMING 
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USE THE CALCULATED HEAD POSITIONS TO 
DETERMINE WHICH PARTICIPANT(S) IS/ARE 
PRESENT IN THE DIRECTION FROM WHICH 
THE SPEECH IS COMING 
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SELECT THE PREVIOUSLY 
IDENTIFIED SPEAKER AS 
THE SPEAKER FOR THE 
CURRENT FRAME 



SELECT EACH 
PARTICIPANT IN THE 
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THE SPEECH IS COMING AS 
A "POTENTIAL" SPEAKING 
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CAMERA(S) AS THE 
CAMERA(S) FROM 
WHICH IMAGE DATA IS 
TO BE STORED 



READ VIEWING PARAMETER FOR THE NEXT 
SPEAKING PARTICIPANT TO DETERMINE AT 
WHOM OR WHAT THEY ARE LOOKING 
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READ HEAD POSITION AND ORIENTATION 
FOR THE SPEAKING PARTICIPANT 
TOGETHER WITH THE HEAD POSITION AND 
ORIENTATION OF THE PARTICIPANT BEING 
SPOKEN TO OR THE POSITION AND 
ORIENTATION OF THE OBJECT BEING 
SPOKEN TO 
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DETERMINE THE CAMERA WHICH BEST 
SHOWS THE SPEAKING PARTICIPANT 
AND THE PARTICIPANT BEING SPOKEN 
TO OR THE OBJECT BEING SPOKEN TO, 

AND SELECT THIS CAMERA AS A 
CAMERA FROM WHICH IMAGE DATA IS 
TO BE STORED 
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READ POSITION AND DIRECTION OF NEXT ^ S1 76 
CAMERA 



CAN THE "\^^S178 
CAMERA SEE BOTH 
THE SPEAKING PARTICIPANT 
AND THE PARTICIPANT OR OBJECT 
AT WHICH THE SPEAKING 
PARTICIPANT IS 
LOOKING? 



NO 



YES 



CALCULATE AND STORE VALUE 
REPRESENTING THE QUALITY OF THE VIEW 
THAT THE CAMERA HAS OF THE SPEAKING 
PARTICIPANT 
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CALCULATE AND STORE VALUE 
REPRESENTING THE QUALITY OF THE VIEW 
THAT THE CAMERA HAS OF THE 
PARTICIPANT OR OBJECT AT WHICH THE 
SPEAKING PARTICIPANT IS LOOKING 



COMPARE THE QUALITY VALUES OF THE 
SPEAKING PARTICIPANT AND THE 

PARTICIPANT OR OBJECT AT WHICH THE 
SPEAKING PARTICIPANT IS LOOKING, AND 
STORE THE VALUE FOR THE WORST VIEW 



COMPARE THE STORED "WORST VIEW" 
VALUES, AND SELECT THE CAMERA WHICH 
HAS THE BEST "WORST VI EW" VALUE AS A 
CAMERA FROM WHICH IMAGE DATA SHOULD 
BE STORED IN THE MEETING ARCHIVE 
DATABASE 
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PRECEDING FRAME BEING CONSIDERED 
FROM THE MEETING ARCHIVE DATABASE 

FOR EACH "POTENTIAL" SPEAKING 
PARTICIPANT WHO IS NOT THE SAME AS 
THE SPEAKING PARTICIPANT FOR THE 



CURRENT FRAME 



FIG. 17 
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FIG. 19 A 



The following parts of the meeting are relevant. Please 
select one for playback: 



1. Speech starting at 10 mins 0 sees (0.4 x full meeting time) 

2. Speech starting at 12 mins 30 sees (0.5 x full meeting time) 



FIG. 19 B 
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REQUEST USER TO INPUT THE NAMES OF ^ S302 
EACH PARTICIPANT 



ALLOCATE A UNIQUE PARTICIPANT NUMBER 
TO EACH PARTICIPANT AND STORE 
RELATIONSHIP BETWEEN PARTICIPANT 
NUMBERS AND NAMES IN THE MEETING 
ARCHIVE DATABASE 
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REQUEST USER TO INPUT DATA DEFINING A 
HEAD MODEL FOR EACH PARTICIPANT FOR 
WHOM A MODEL IS NOT ALREADY STORED 
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STORE THE DATA INPUT BY THE USER 
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RENDER EACH MODEL INPUT BY THE USER 
TO DISPLAY THE MODEL AND REQUEST THE 
USER TO IDENTIFY FEATURES IN EACH 
MODEL 
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REQUEST USER TO INPUT THE SPEECH 
RECOGNITION PARAMETERS FOR EACH 
PARTICIPANT FOR WHOM THE PARAMETERS 
ARE NOT ALREADY STORED 
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STORE THE SPEECH RECOGNITION 
PARAMETERS INPUT BY THE USER 
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REQUEST THE USER TO PERFORM STEPS 
TO ENABLE THE CAMERA TO BE 
CALIBRATED 
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PERFORM PROCESING TO CALIBRATE THE 
CAMERA AND STORE CAMERA CALIBRATION 
DATA 
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REQUEST THE NEXT PARTICIPANT TO SIT 
DOWN 
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WAIT FOR PREDETERMINED PERIOD OF 
TIME 
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PROCESS IMAGE DATA TO DETERMINE AN 
ESTIMATE OF THE POSITION OF THE 
SEATED PARTICIPANT'S HEAD 



S330 



FIT THE PARTICIPANT'S HEAD MODEL TO 

THE IMAGE DATA TO DETERMINE AN 
ESTIMATE OF THE ORIENTATION OF THE 
PARTICIPANT'S HEAD 
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FIG. 23 (cont) 



® 



TRACK THE HEAD OF THE PARTICIPANT 



o 

m 
w 
ry 

m 
w 
w 



S334 




YES 



NO 



OUTPUT AUDIBLE SIGNAL TO INDICATE 
THAT MEETING CAN COMMENCE 
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POSITION AND ORIENTATION OF THE 
PARTICIPANT'S HEAD 
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RENDER THE MODEL OF THE PARTICIPANT'S 
HEAD USING THE CURRENT ESTIMATES AND 
THE CAMERA CALIBRATION DATA 
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EXTRACT CAMERA IMAGE DATA FROM AN 
AREA SURROUNDING THE EXPECTED 
POSITION OF EACH HEAD FEATURE 
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PERFORM MATCHING OF THE RENDERED 
HEAD MODEL AND CAMERA IMAGE DATA TO 
FIND THE CAMERA IMAGE DATA WHICH 
BEST MATCHES THE RENDERED HEAD 
MODEL 
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DETERMINE THE 3D POSITION AND 
ORIENTATION OF THE PARTICIPANT'S 
HEAD FOR THE CURRENT FRAME OF 
IMAGE DATA 
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INPUT THE POSITIONS OF THE HEAD 
FEATURES IN THE CAMERA IMAGE 
DATA INTO A KALMAN FILTER TO 
GENERATE AN ESTIMATE OF THE 3D 
POSITION AND ORIENTATION OF THE 
HEAD FOR THE NEXT FRAME OF 
IMAGE DATA 
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DETERMINE A VIEWING PARAMETER 
FOR EACH PARTICIPANT DEFINING AT 
WHOM THE PARTICIPANT IS LOOKING 
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IS SPEAKING 
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STORE THE VIEWING PARAMETER FOR THE 
SPEAKING PARTICIPANT FOR SUBSEQUENT 
ANALYSIS 
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PERFORM SPEECH RECOGNITION 
PROCESSING USING THE SPEECH 
PARAMETERS OF THE SPEAKING 
PARTICIPANT TO GENERATE TEXT DATA 



ENCODE THE VIDEO DATA AND SOUND DATA 
AS MPEG 2 DATA AND STORE IN THE 
MEETING ARCHIVE DATABASE 
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STORE THE VIEWING PARAMETER FOR 
EACH PARTICIPANT IN THE MEETING 
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PROCESS THE STORED VIEWING 
PARAMETER VALUES FOR THE PARTICIPANT 
WHO HAS STOPPED SPEAKING TO 
GENERATE A VIEWING HISTOGRAM 
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STORE THE VIEWING HISTOGRAM IN THE 
MEETING ARCHIVE DATABASE, LINKED TO 
THE ASSOCIATED TEXT 
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EACH NOTIONAL LINE CONNECTING THE 
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DETERMINE THE NUMBER OF OCCURENCES 
OF EACH VIEWING PARAMETER VALUE 
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WITH THE HIGHEST NUMBER OF 
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IDENTIFY THE PARTICIPANT DEFINED BY 
THE SELECTED VIEWING PARAMETER 
VALUE 
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The following parts of the meeting are relevant. Please 
select one for playback: 



1. Speech starting at 10 mins 0 sees (0.4 x full meeting time) 

2. Speech starting at 12 mins 30 sees (0.5 x full meeting time) 



FIG. 35 B 
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